Neural Network Structures and Isomorphisms: Random Walk Characteristics of the Search Space

In this article we deal with a quite general topic in evolutionary structure optimization, namely redundancy in the encoding due to isomorphic structures. This problem is well known in topology optimization of neural networks (NNs). In the context of structure optimization of NNs we observe similar phenomena of rare and frequent structures as are known from molecular biology. The degree to which isomorphic structures, i.e., classes of equivalent NN topologies, enlarge the search space depends on the restrictions of the allowed structures and on the representation of the search space. For restricted network topologies, like NNs with a maximum number of layers, some properties can be analyzed analytically. For more general structures we estimate the characteristics of the search space using data stemming from random walks. For restricted NN topologies, the search process is affected by isomorphic structures. However, in the absence of restrictions, the search space becomes so large that the bias induced by isomorphisms can be neglected.